Do We Know What LLMs Don’t Know? Consistency in Knowledge Probing

Evaluating the Robustness and Reliability of Knowledge Gap Probes in Large Language Models

Published

May 27, 2025

Authors: R. Zhao et al.
Published on Arxiv: 2025-05-27
Link: http://arxiv.org/abs/2505.21701v2
Institutions: LMU Munich • Munich Center for Machine Learning (MCML)
Keywords: large language models, knowledge probing, model consistency, knowledge gaps, hallucinations, abstention, prompt sensitivity, robustness, MMLU, Hellaswag, calibration, self-consistency, decision metrics

Random Unsplash-style image

Large language models (LLMs) have become widely used but suffer from hallucinations—generating fluent yet incorrect outputs. Identifying areas where LLMs lack knowledge is critical for building trust and reliability in their applications. While various knowledge probing methods exist, little is known about the robustness and reliability of these methods themselves.

To address this gap, the authors propose a new framework and set of metrics for systematically studying the consistency of knowledge probing methods in LLMs. Their main approach and contributions include:

Their experiments yield important empirical findings:

Building on these results, the authors draw several key conclusions: